{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将解析文本的常见预处理步骤。这些步骤通常包括：\n",
    "1. 将文本作为字符串加载到内存中。\n",
    "2. 将字符串拆分为词元（如单词和字符）。\n",
    "3. 建立一个词表，将拆分的词元映射到数字索引。\n",
    "4. 将文本转换为数字索引序列，方便模型操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import re\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# 文本总行数: 3221\n",
      "the time machine by h g wells\n",
      "twinkled and his usually pale face was flushed and animated the\n"
     ]
    }
   ],
   "source": [
    "d2l.DATA_HUB['time_machine'] = (d2l.DATA_URL + 'timemachine.txt',\n",
    "'090b5e7e70c295757f55df93cb0a180b9691891a')\n",
    "def read_time_machine(): #@save\n",
    "    # \"\"\"将时间机器数据集加载到文本行的列表中\"\"\"\n",
    "    with open(d2l.download('time_machine'), 'r') as f:\n",
    "        lines = f.readlines()\n",
    "    # \"\"\"匹配非A-Za-z的字符，替换成空格\n",
    "    # 加号的作用是将连续的非字母字符视为一个整体。例如，1234 被视为一个匹配，因为它是由多个数字组成的\n",
    "    # \"\"\"\n",
    "    return [re.sub('[^A-Za-z]+', ' ', line).strip().lower() for line in lines]\n",
    "lines = read_time_machine()\n",
    "print(f'# 文本总行数: {len(lines)}')\n",
    "print(lines[0])\n",
    "print(lines[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词元化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'time', 'machine', 'by', 'h', 'g', 'wells']\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "['i']\n",
      "[]\n",
      "[]\n",
      "['the', 'time', 'traveller', 'for', 'so', 'it', 'will', 'be', 'convenient', 'to', 'speak', 'of', 'him']\n",
      "['was', 'expounding', 'a', 'recondite', 'matter', 'to', 'us', 'his', 'grey', 'eyes', 'shone', 'and']\n",
      "['twinkled', 'and', 'his', 'usually', 'pale', 'face', 'was', 'flushed', 'and', 'animated', 'the']\n"
     ]
    }
   ],
   "source": [
    "def tokenize(lines,token='word'):\n",
    "    \"\"\"将文本行拆分为单词或字符单元\"\"\"\n",
    "    if token=='word':#单词\n",
    "        return [line.split() for line in lines]\n",
    "    elif token=='char':#单个字符\n",
    "        return [list(line) for line in lines]\n",
    "    else:\n",
    "        print(\"error\")\n",
    "\n",
    "tokens=tokenize(lines)\n",
    "for i in range(11):\n",
    "    print(tokens[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocab类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个 Vocab 类的主要功能是构建一个文本词表，统计词元的频率，并为每个词元分配一个唯一的索引。它可以用于自然语言处理任务中的文本预处理和模型输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Vocab: #@save\n",
    "    \"\"\"文本词表\"\"\"\n",
    "    #tokens：词元列表\n",
    "    # min_freq:词元最小频率，低于此频率会被过滤\n",
    "    # reserved_tokens：可选参数，表示保留的特殊词元（如 <pad>、<bos> 等）。\n",
    "    def __init__(self, tokens=None, min_freq=0, reserved_tokens=None):\n",
    "        if tokens is None:\n",
    "            tokens = []\n",
    "        if reserved_tokens is None:\n",
    "            reserved_tokens = []\n",
    "        \n",
    "        # 按出现频率排序\n",
    "        counter = count_corpus(tokens)\n",
    "        self._token_freqs = sorted(counter.items(), key=lambda x: x[1], reverse=True)\n",
    "        \n",
    "        # 未知词元的索引为0\n",
    "        self.idx_to_token = ['<unk>'] + reserved_tokens\n",
    "        self.token_to_idx = {token: idx for idx, token in enumerate(self.idx_to_token)}\n",
    "        \n",
    "        # 过滤低频词元\n",
    "        for token, freq in self._token_freqs:\n",
    "            if freq < min_freq:\n",
    "                break\n",
    "            if token not in self.token_to_idx:\n",
    "                self.idx_to_token.append(token)\n",
    "                self.token_to_idx[token] = len(self.idx_to_token) - 1\n",
    "    # 返回词表的大小\n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)\n",
    "\n",
    "\n",
    "    \"\"\"\n",
    "    根据词元获取其索引。如果输入是单个词元，则返回其索引；如果是列表或元组，则返回对应的索引列表。\n",
    "    \"\"\"\n",
    "    def __getitem__(self, tokens):\n",
    "        if not isinstance(tokens, (list, tuple)):\n",
    "            return self.token_to_idx.get(tokens, self.unk)\n",
    "        return [self.__getitem__(token) for token in tokens]\n",
    "\n",
    "    @property\n",
    "    def unk(self): # 未知词元的索引为0\n",
    "        return 0\n",
    "    \n",
    "    @property\n",
    "    def token_freqs(self):\n",
    "        return self._token_freqs\n",
    "    \n",
    "def count_corpus(tokens): #@save\n",
    "    \"\"\"统计词元的频率\"\"\"\n",
    "# 这里的tokens是1D列表或2D列表\n",
    "    if len(tokens) == 0 or isinstance(tokens[0], list):\n",
    "        # 将词元列表展平成一个列表\n",
    "        tokens = [token for line in tokens for token in line]\n",
    "    return collections.Counter(tokens)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们首先使用时光机器数据集作为语料库来构建词表，然后打印前几个高频词元及其索引"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('<unk>', 0), ('the', 1), ('i', 2), ('and', 3), ('of', 4), ('a', 5), ('to', 6), ('was', 7), ('in', 8), ('that', 9)]\n"
     ]
    }
   ],
   "source": [
    "vocab = Vocab(tokens)\n",
    "print(list(vocab.token_to_idx.items())[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，我们可以将每一条文本行转换成一个数字索引列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文本: ['the', 'time', 'machine', 'by', 'h', 'g', 'wells']\n",
      "索引: [1, 19, 50, 40, 2183, 2184, 400]\n",
      "文本: ['twinkled', 'and', 'his', 'usually', 'pale', 'face', 'was', 'flushed', 'and', 'animated', 'the']\n",
      "索引: [2186, 3, 25, 1044, 362, 113, 7, 1421, 3, 1045, 1]\n"
     ]
    }
   ],
   "source": [
    "for i in [0, 10]:\n",
    "    print('文本:', tokens[i])\n",
    "    print('索引:', vocab[tokens[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# #一个小技巧，从tokens提取每一行为line，再从line中提取每个单元，赋予索引\n",
    "# tokens = [\n",
    "#     [\"I\", \"love\", \"programming\"],\n",
    "#     [\"Programming\", \"is\", \"fun\"],\n",
    "#     [\"I\", \"love\", \"learning\"]\n",
    "# ]\n",
    "# [vocab[token] for line in tokens for token in line]\n",
    "# [0, 1803, 0, 0, 38, 0, 0, 1803, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 整合所有功能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(170580, 28)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def load_corpus_time_machine(max_tokens=-1):\n",
    "    # 返回时光机器数据集的词元索引表和列表\n",
    "    lines=read_time_machine()\n",
    "    tokens=tokenize(lines,'char')\n",
    "    vocab=Vocab(tokens)\n",
    "    # 因为时光机器数据集中的每个文本行不一定是一个句子或一个段落，\n",
    "    # 所以将所有文本行展平到一个列表中\n",
    "    corpus=[vocab[token] for line in tokens for token in line]\n",
    "    if max_tokens>0:\n",
    "        corpus=corpus[:max_tokens]\n",
    "    return corpus,vocab\n",
    "corpus,vocab=load_corpus_time_machine()\n",
    "len(corpus),len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## torchtext库 入门"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 添加保留词元"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<\\pad>：用于填充，通常在处理变长序列时使用，以确保所有序列的长度一致。<br>\n",
    "<\\unk>：表示未知词元，通常用于替代那些不在词表中的词。<br>\n",
    "<\\bos>：句子开始标记，通常在序列生成任务中使用，表示句子的开始。<br>\n",
    "<\\eos>：句子结束标记，表示句子的结束。<br>\n",
    "【正常没有\\】<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一些其他常用库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data import BucketIterator #读取数据集的\n",
    "from torchtext.data import Field\n",
    "from torchtext.data import TabularDataset #可以读json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer=lambda x: x.split()#分词\n",
    "\"\"\"【定义文本字段】句子读取\"\"\"\n",
    "# 开始标记，句子结束标记，表示字段为序列，使用词汇表，指定分词器，设置文本为小写\n",
    "quote=Field(init_token=\"<sos>\",eos_token=\"<eos>\",sequential=True,use_vocab=True\n",
    "            ,tokenize=tokenizer,lower=True\n",
    "            )\n",
    "\"\"\"分数（标签）\"\"\"\n",
    "# sequential=False代表不是序列而是一个数值\n",
    "score=Field(sequential=False,use_vocab=False)\n",
    "\"\"\"数据加载指定\"\"\"\n",
    "# 在数据集中，字段quote将变指代为q，字段score被指代为s\n",
    "fields={\"quote\":('q',quote),\"score\":(\"s\",score)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**制作数据集**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ps:json每个字典放同一行，不然会报错"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch.q:tensor([[ 2,  2],\n",
      "        [29, 35],\n",
      "        [31, 21],\n",
      "        [ 9, 26],\n",
      "        [28, 16],\n",
      "        [20, 17],\n",
      "        [ 4, 36],\n",
      "        [27, 34],\n",
      "        [ 3, 33],\n",
      "        [ 1, 18],\n",
      "        [ 1, 22],\n",
      "        [ 1, 24],\n",
      "        [ 1, 14],\n",
      "        [ 1,  7],\n",
      "        [ 1, 10],\n",
      "        [ 1,  3]])\n",
      "batch.s:tensor([0, 1])\n",
      "batch.q:tensor([[ 2],\n",
      "        [12],\n",
      "        [23],\n",
      "        [ 6],\n",
      "        [ 5],\n",
      "        [ 8],\n",
      "        [13],\n",
      "        [19],\n",
      "        [ 6],\n",
      "        [ 5],\n",
      "        [32],\n",
      "        [30],\n",
      "        [ 7],\n",
      "        [15],\n",
      "        [ 4],\n",
      "        [11],\n",
      "        [25],\n",
      "        [ 3]])\n",
      "batch.s:tensor([1])\n"
     ]
    }
   ],
   "source": [
    "# 读取数据集\n",
    "train_data,test_data=TabularDataset.splits(path=\"./\",train='test.json',validation=\"test.json\",\n",
    "                                           format='json',fields=fields\n",
    "                                           )\n",
    "# 构造词汇表(数据，词汇表最大大小，词元最小频率)\n",
    "quote.build_vocab(train_data,max_size=10000,min_freq=1)\n",
    "# 训练数据迭代器，和测试数据迭代器\n",
    "train_iter,test_iter=BucketIterator.splits((train_data,test_data),batch_size=2)\n",
    "for batch in train_iter:\n",
    "    print(f\"batch.q:{batch.q}\")\n",
    "    print(f\"batch.s:{batch.s}\")\n",
    "#每一列是一句话"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<unk>', '<pad>', '<sos>', '<eos>', 'a', 'for', 'pray', 'to', 'an', 'and']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quote.vocab.itos[:10]#查看字典的映射"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "索引： 0 单词： <unk>\n",
      "索引： 1 单词： <pad>\n",
      "索引： 2 单词： <sos>\n",
      "索引： 3 单词： <eos>\n",
      "索引： 4 单词： a\n",
      "索引： 5 单词： for\n",
      "索引： 6 单词： pray\n",
      "索引： 7 单词： to\n",
      "索引： 8 单词： an\n",
      "索引： 9 单词： and\n",
      "索引： 10 单词： blame.\n",
      "索引： 11 单词： difficult\n",
      "索引： 12 单词： do\n",
      "索引： 13 单词： easy\n",
      "索引： 14 单词： else\n",
      "索引： 15 单词： endure\n",
      "索引： 16 单词： everything\n",
      "索引： 17 单词： in\n",
      "索引： 18 单词： is\n",
      "索引： 19 单词： life,\n",
      "索引： 20 单词： like\n",
      "索引： 21 单词： must\n",
      "索引： 22 单词： no\n",
      "索引： 23 单词： not\n",
      "索引： 24 单词： one\n",
      "索引： 25 单词： one.\n",
      "索引： 26 单词： own\n",
      "索引： 27 单词： potato!\n",
      "索引： 28 单词： rice\n",
      "索引： 29 单词： stand\n",
      "索引： 30 单词： strength\n",
      "索引： 31 单词： tall,\n",
      "索引： 32 单词： the\n",
      "索引： 33 单词： there\n",
      "索引： 34 单词： world.\n",
      "索引： 35 单词： you\n",
      "索引： 36 单词： your\n"
     ]
    }
   ],
   "source": [
    "# 查看单词索引和对应的词\n",
    "for i in range(len(quote.vocab)):\n",
    "    print('索引：',i,\"单词：\",quote.vocab.itos[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"单词转数字\"\"\"\n",
    "quote.vocab.stoi[\"and\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quote.vocab.stoi[\"love\"]#未收录的词，对应UNK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "NLP",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.21"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
